« C r O II T 



R e s u M e t 



ED OH H6 



CA 000 9R2 



CmTCRIA FOR NETH0C0L06ICAL ADEQUACY FOR RESEARCH OM 
EDUCATIONAL CHANCE. 
iY- CEPHART. WILLIAN J. 



CDRS PRICE NF-SO.iS HC-S2.40 SOP. 



PUB DATE SEP jS5 



descriptors- ^educational CHANCEf ^EDUCATIONAL RESEARCH. 
techniques. ^RESEARCH NETHODOLOCY. MODELS. EVALUATION. 
RESEARCH. MILWAUKEE 

research adequacy must be assessed and standards drawn 

IF PROCRECC IS TO BE MADE IN THE ACCUMULATION OF KNOWLEDCE. 
this DISCUSSIOfT OF METHODOLOGICAL CRITERIA FOCUSES UPON THE 
FOLLOWING TOPICS — (1) A LOGIC FRAMEWORK FOR EDUCATIONAL 
research. (2) GENERAL CRITERIA FOR RESEARCH EVALUATION. (3) 
ELEMENTS OF THE STUDY OF THE EDUCATIONAL CHANGE PROCESS. (4) 
METHODS AND TECHNIQUES FOR STUDYING THE CHANGE PROCESS 
COMPONENTS. AND (5) CRITERIA OF ADEQUACY FOR EVALUATING 
RESEARCH TECHNIQUES IN THE STUDY OF EDUCATIONAL CHANGE. (GB) 









CRITERIA FOR METHODOLOGICAL ADEQUACY FOR 
RESEARCH ON EDUCATIONAL CHANGE 



By 






U 



William J. Gephart 

Director of Research and Experimentation 
University of Wiscons in-Mi Iwaukee 









September 1365 



I j. HMimm « nin. Diaiin ( VBFAK 

tHKEVENaim 

nBNoiim 

mMHMfttramtwnnBii. rans«KWMONMB 



\aA 000 



1 er|c| 






CRITERIA FOR METHODOLOGICAL ADEQUACY FOR 
RESEARCH ON EDUCATIONAL CHANGE 

The development of criteria for methodological adequacy 
of educational research has been a problem faced by professional 
educators for almost fifty years. In any absolute sense it is 
and should remain an unsolved problem as the field anticipates 
the evolution of improvements in research strategies, method- 
ologies, and techniques. Thus, a statement of criteria for 
evaluating research is of value only within specific temporal 
boundaries, yet at the same time specification of evaluative 
criteria is highly necessary. Unless a systematic assessment 
of the strengths and weaknesses of existing research on the edu- 
cational change process is undertaken, two deterrents to progress 
exist. Firstj since individual research efforts vary in adequacy, 
"facts" generated by these studies vary in value. Second, the 
development of improved strategies, methods, and techniques for 
research on educational change rests* heavily on the analysis 
of existing, techniques. Both of these are stumbling blocks to 

-r N. 

the continued accumulation of knowedge necessary for "growing" 
the "inductive inference tree" described by Flatt as crucial to 
advancement in a substantive area.^ 

^J. R. Platt, "Strong Inference," Science. 146:347-52: 
October, 1964. 
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The evaluative criteria presented in this paper have 
evolved from two sources, literature and research on the re- 
search process. Much of the literature on the research pro- 
cess exists in the form of textual materials which contain the 
rationale and elaboration for the evaluative criteria presented 
here. There is also an expanding body of research literature in 
which research is the substantive topic. Six discrete direc- 
tions can be observed in this literature. 

1. The identification of type and frequency 
of errors found in educational research.^.' 

2. The assessment of the content and form of 
research reports.^ 

3- The assessment of the value of research 
through study of its impact on textual * 
materials.^ 

The identification of type and frequency 
of inadequacies in research proposals.^ 



1 

/ 



V r* The Elements of Research: Revised Ed it inn 

Inc., 1942. p. *,5-7. Whitney present! 
n tabular form lists by eight authors in which., research errors 
m attitude, method, and technique are identified. The summa- 
rized papers span a period from I9I9 to I930. 

3 

G. M. Wilson, "Research; Suggested Standards for Summa- 
Reporting Applied to T^o Recent Sunmiaries of Studies 
in Arithmetic. Journal of Educational Rese arch 28*187-44 

November 1934. — 

4 

Resea rch°al!HM!l ^' 'The Relationship Between Arithmetic 

Research and the Content of Arithmetic Textbooks (1900-1957) '• 
The Arithm etic Teacher 7:178-ft^- April locn ■ 

5 

G. R. Smith, "Inadequacies in a Selected Sample of Research 

oposals. Unpublished doctoral dissertation. Teachers College 
Columbia University, 1964. -v.iics 1,01 lege. 
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5. The analysis of the role of theory in the 
literature on the research process. 

6 . The identification of the techniques, methods, 
or designs employed in educational research.^ 

Since the latter type of study — the use of judges to eval- 
uate the adequacy of research — relates most directly to the cur- 
rent project, an expanded discussion is presented. Two general 
approaches to the use of expert judges in evaluating research 
adequacy seem to be employed. One group of studies involves the 
identification of one or. a number of eminently qualified persons 
and asking them to evaluate selected research. The second approach 
also involves the selection of qualified persons but asks them to 
employ some specified evaluative criteria. Examples of the pro- 
duct of the unstructured approach can be seen in the research 

evaluation contained in the Review of Educational Research , a 

8 

study of research on counseling and guidance, a study of research 

9 

in teacher education. The structured approach is illustrated in 



K. E. Lake, "Inductive Methodology Versus Hypothetic- 
Deductive Methodology in Educational Research." Unpublished 
doctoral dissertation. University of Kansas, I 96 I. 

H Bixler, "Check Lists for Educational Research," 

New York: Teachers College, Columbia University, 1928. p. 85-7. 

8 

W. B. King, Survey of the Status of Research in Guidance 
and Counsel ing . Washington, D.C.: U.S.O.E. Cooperative Research 

Project Number F-1, 1962. 

9 

F. R. Cyphert and E. Spaights, An Analysis and Projection 
of Research in Teacher Education . Washington, D.C.: U.S.O.E. 

Cooperative Research Project Number F-015, 1964. 
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the work of Johnson,'^ the American Institute of Research,'' and 
12 

Gephart. 

The .-orrent project attempts the synthesis of these two 
approaches, as a statement of criteria for. evaluating research on 
educational change has been employed by the project staff, and 
further unstructured evaluation by persons selected for specific 
competencies is scheduled during the conference. 

The discussion which follows will focus on the criteria 
of methodological adequacy. To do so, it will treat sequentially 
the following topics; 

1 . 



2 . 

3. 

k. 

5. 



A . plausible logic framework for educational 
research. 

General criteria for research evaluation. 

Elements of the study of the change process. 

Methods and techniques for studying the change 
process components. 

Criteria of adequacy for these techniques. 



G. B. Johnson, "A Method for Evaluating Research Articles 
in Education." Journal of Educational Research SI : l4q-«il • 

October, I 957 

'^American Institute of Research, "A Procedure for Evaluating 
Graduate Research on the Basis of the Thesis." Pittsburg: October 

1955. 



12 

W. J. Gephart, Development of an Instrument for Evaluating 
Reports o f Educational Research. Washington, O.C.: U.S.O.E. 

Cooperative Research Project Number S-014, 1964. 
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A PLAUSIBLE LOGIC FRAMEWORK FOR EDUCATIONAL RESEARCH 

The literature on the research process presents consider- 
able agreement regarding the components of the research process. 

The student of the research process has little difficulty identi- 
fying components of (1) problem identification and development, 

(2) evolution of hypotheses, (3) evaluation and synthesis of pre- 
vious research, (4) designing the specific study, (5) analyzing the 
data accumulated, and (6) derivation of the conclusions and impli- 
cations. Although understanding of each of these is important to 
the conduct of research and to its evaluation, the discussion which 
follows focuses on the premise that each research effort is in 
itself a logical argument.. 

As knowledge builds up about a substantive area, each piece 
of research attempts to provide some direction for further expan- 
sion of knowledge. That is, when an unknown or a problem is en- 
countered, several possible solutions are identified, each of 
which presents an hypothesis to explain the unknown or solve the 
problem. The tests of these hypotheses assist in the most effi- 
cient movement to the next "fork in the tree." 

The hypothesis in a given study then is "a conjectural state- 

14 

ment about the relationship between two or more variables." The 

R. Platt, o£. cit . 

14 

F.N. Kerlinger, Foundations of Behavioral Research . 

New York: Holt, Rinehart and Winston, Inc., 1964. 







focus of each study is the establishment of the truth of the hypoth- 
esis. Does empirical evidence support the validity of a theoreti- 
cally evolved hypothesis? 

This reasoning form differs from formal logic where observa- 
tions regarding the truth of. an antecedent are used to infer truth 
of a consequen4i»^ For example: 

Major Premise: 



Minor Premise: 
Cone 1 us ion: 



If I live in Oconomowoc (antecedent); 
then I 1 I ve in Wisconsin (consequence) 

(a) I live in Oconomowoc, or 

(b) I do not live in Oconomowoc. 

(a) The consequence is true, | 1 ive 
in Wisconsin, or, 

(b) No conclusion regarding:* the con- 

sequences. 

If forced to observe on the consequence, the possibility of a posi- 
tive conclusion is removed. 

Minor Premise: (a) | live in Wisconsin, or 

(b) I do not live in Wisconsin. 

Conclusion: (a) No conclusion regarding the ante- 

cedent, or, 

(b) The antecedent is false, | do 
not live in Oconomowoc . 

Hypotheses in the social sciences are generally not directly 
observable. Thus, the researcher is compelled to consider the 
hypothesis as the antecedent in a syllogistic major premise which, 
if a true statement, would result in certain observable conse- 
quences. The form of logical inference suggested by the mathemat ic iai 




Polya, is appropriate to infer an answer to the question, "Is 
the hypothesis true?" 

Major Premise: If A (hypothesis) then B (consequence) 

a true statement. 

Minor Premise: B (the consequence) is observed. 

Conclusion: The truth of A is supported. 

In proposing the application of this "plausible inference 
pattern" to educational research, Raths indicates the need for 
the insertion of a qualification pror to the minor premise. 

This qualification is necessitated due to awareness that var- 
iables other than those specified in the hypothesis may affect 
the degree to which the consequences are observed. For example, 
performance at a learning task may be due to: age, sex; prior 

knowledge, attitudes, etc., as well as— or even rather than— 
being due to a specific treatment. In the research process this 
qualification is evident in the form of control of such extra- 
neous and error variances. Thus, the inference pattern skeleton 
of the research process is outlined as follows: 



Major Premise: If A (hypothesis) then B (consequence) 

is a trus statement. 

Qualification: B occuring without A being true 

IS hardly credible due to con- 
trols employed. (Amount of control 
can be equated to the number of alter- 
native hypotheses eliminated.) 



George Polya, presented in an unpublished lecture. 
University of Wi scons in-Mi Iwaukee, December, 1963. 

Raths, Unpublished paper presented at the American 
Educational Research Assoc i cat ion Annual Conference, February 
1964. 
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Minor Premise: B (consequence) is observed. 

Conclusion: The truth of A is strongly supported. 

(The strength of the support is pro- 
portional to the amount of control.) 

The components of this inference pattern relate to the 
components of the research process. The researcher through the 
identification, definition, and delimitation of a problem, devel- 
ops a theory consisting of what is known and what is suspected. 

The latter need to be tested in the form of hypotheses. The 
determination of the data necessary to test the hypothesis is 
arrived at by deducing the consequences which may be observed 
if the hypothesis is true. Thus, through the problem, hypothesis, 
related research components of the research process, the major pre- 
mise is established. The design and data analysis components of 
the research process equate to the qualification, in that design 
generally refers to the plans made to ensure the collection of the 
most irelevant data on the consequence as a test of the hypothesis 
and data analysis techniques are also employed as controls'. The 
findings resulting from the analysis of the data are the specifics 
of the minor premise,. and finally, the conclusion component of 
the research process coincides with the conclusion in the plaus- 
ible inference pattern. 

GENERAL CRITERIA FOR RESEARCH EVALUATION 
Criteria for evaluating research have been stated by numer- 
ous individuals. These may be listed in two categories: lack- 

ing or having data available regarding val idi ty and/or reliability. 
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Examples of checklists lacking such. data are those proposed by " 
SymondsJ^ Van Dalen,*® Farquahar and KrumboltzJ^ and Mouly.^° 
Those proposed by the American Institute of- Research, Johnson, 
Wandt,^^ and Gephart^^ have endured some empirical assessment. 
However, simil^^r to any group of psychological measuring devices, 
they seem to differ in quality for the purpose at hand. With 
the exception of Wandt's work which is yet to be reported, the 
instruments in this latter group are critiqued below. 

Johnson's instrument is exceedingly brief.' It consists 
of eleven items: two each for evaluating the problem, materials, 

and subjects; three for the method of procedure; and one each 
for evaluating results and conclusions. The nature of the items 
further reduces the value of Johnson's instrument. For example, 
about the problem the instrument asks, "Is it clear? 1-2-3-4-5." 

Symonds, Research Checklist in Educational Psychol- 
ogy." Journal- of Educational Psychol oov 47:100-9: February 1956. 

B. Van Dalen, "Research Checki ist in Education." 
Educational Acfaiinistration and Supervision . 44:174-81; May 1958. 

19 

W. W. Farquahar and J. D. Krumbo1tz,"A Checklist for 
Evaluating Experimental Research in Psychology and Education." 
Journal of Educational Research 52:534-5; September 1959. 

20 

G. J. Mouly. The Science of Educational Research. New 
York: American Book Company, I 963 . p. 503-4. 

21 

'American Institute of Research, o£. cit . 

22 

G. B. Johnson, Jr, o£. cit . 

23 . . 

E.* Wandt is chairman of an ad hoc committee on research 
evaluation for the American Educational Research Association. 

24 

W. J. Gephart, o£. cit . 
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The focus of a judgment on a term such as '^lear'.' is asking for 
the application of a- rubber yardstick. What is clear to one per- 
son may be obtuse to another. The other items focus on terms of 
the same nature: "significance," "authoritative sources," "large 

enough samples," "adequate," 'V>rder1y and systematic," "proper and 
modern techniques," etc. 

Johnson %ifas able to obtain significant agreement tdien four- 
teen of his students (late in a course on educational research) 
and four of his colleagues used his evaluative instrument. Pri- 
vate correspondence with Johnson indicates that the course was 
devoted to the definition of the terms cited above. Thus, it 
would seem that when commonality of definition of the research 
process exists, agreement among evaluatprs can be obtained in 
assessing research adequacy. It is doubtful though that this 
brief instrument can provide the necessary definition. 

The project reported by the American Institute of Research 
(AIR) substantiates the conclusion drawn from Johnson's wrk.^^' 
Although the instrument in this study is considerably longer, it 
still uses terminology which needs definition. The AIR instru- 
ment is a distinct improvement in the responses requested of the 
rater in that it lists actions to be completed in the research 
process and requests timo responses; did the action occur; and, 
does that occurrence contribute or detract from the value of the 
research* 

25 

6. B. Johnson, 0£. cit . 
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26 

Gephart, following some work with Clark, Guba, and Smith, 
also focuse'd upon actions inherent in the research process. Nu- 
merous texts vte re analyzed to identify actions to be taken in the 
research process. An attempt was made to avoid the undefinable 
terms such as were found in the above mentioned studies. The re- 
sponse pattern of the AIR instrument was employed. That is, oc- 



currence and value ratings were requested for each item. A 
Cooperative Research Program Grant made possible the use of ten 
competent judges in the establishment of (1) the applicability of 
each item, (2) the comprehensiveness of the instrument, and (3) 
the interrater reliability for evaluations of reports of research 
in professional journals. Significant agreement was found both 
within and among the five jurors who were research design and 
methodology experts and the five substantive experts. 

These efforts seem to imply at least that when judges of 
research adequacy employ the same set of definitions of the re- 
search process, there is reliability. They can agree on the ade- 
quacy of a specific research. It has been suggested that, ««hen jurors 
employing these instruments have sufficient commonality of ex- 
perience and training, the instrument serves as a reminder func- 
tion, calling to the evaluator's mind all of the factors to be 




Gephart, o£. cit . 



27 

'David L. Clark, Egon G. Guba, and Gerald R. Smith, 
"Functions and Definitions of Functions of a Research Proposal 
or Research Report." Unpublished mimeo, Columbus, Ohio, The 
Ohio State University, 1962. 
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considered. If such is the case, .assessment of research -adequacy 
could proceed using Symond's or Van Oalen's checklists or the in- 
struments developed by the AIR or Gephart, for all of them attempt 
comprehensive coverage of the research process. 

Despite differences in terminology, type of response and 
spec i f i c focus of the i nd i v i dua 1 i terns i n the above checki i s ts , 
they have in common the research process components found in every 
text on the research process. All indicate that the evaluation 
task should focus on (1) the problem studied, (2) the hypotheses 
tested or questions asked, (3.) the related literature surveyed, 

(4) the design of the study, (5) the analysis of the data, and 
(6) the conclusions and implications drawn from the study. Thus, 

- it is proposed that the assessment of the adequacy of research on 

change in education should be based on (1) the general criteria 

/ 

of research adequacy, and (2) the criteria relevant to research 
activities specific to techniques of research on change. The 
enumeration of general criteria is presented next. Criteria hav- 
ing relevance only to research on change will be presented after 
brief statements on the elements, methods, and techniques for 
such study. 



o 
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Evaluative Criteria for the Problem -Component 

In discussing the problem component, the literature on the 
research process evidences some degree of agreement on the follow- 
ing activities: 

1. The establishment of the existence of a 
problem. 28. 29 

2. The identification of the factors or variables 
inherent in the problem. 38, 31 

-3. The relating of the problem to its antecedents. 32 > 33 

k. The identification of the limits in the study 

of the problem.^^» 35 



David L. Clark, Egon G. Guba, and Gerald R. Smith, o£. cit . 

29 

G. D. HcGrath, James J. Jel inek, and Raymond E. Wochner, 
Educational Research Methods. New York: The Ronald Press Company, 

1963. p. 2k 

30 

George J. Mouly, o£. cit . 

31 

Debold B. Van Dalen, Understanding Educational Research . 

New York: HcGraw-Kill Book Company, 1962. p. 23. ^ ^ ^ 

32 

G. D. HcGrath, James J. Jel inek, and Raymond E. Wochner, 

OP . cit . p. 52. 

33 . 

David L. Clark, Egon G. Guba, and Gerald R. Smith, o£. cit . 
3k 

G. D. HcGrath, J. J. Jel inek, and R. E. Wochner, op. cit . 
p. 52. 

^^D. B. Van Dalen^ o£. cit . p. 52. 
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5. 

6. 

7. 



tte': *s«areh .'32,°37*l8f ‘f§ of 

tte ‘«™!nology utilized in 

>t is proposed that these seven points can hesynthe- 
sued into four criteria for evaluating the prohie™. 

'■ o“r: — nee 

cXrf?::e^r;„f rhi^o^hfet^’' 

X%\te7tud;'ucon*2^^^ "m its within 



New Vorh:“'i:v^;-McX®’c^^^f2l^^ 
37ft - 



38 



0. B. VanOalen, ofi. cU. p.,125. 



39: • • 

L* Clark, E. g. Cuba, and G R Smii-k 
40 og. cit . 



40 

4r 



Ibid. 



Ndw Jersey; Prent 1ce- H^^^[nc. ^"9lewpod. Cl iffs. 
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Several types of situations are described which should be 
helpful in deciding the establishment of the existence of a 

I 

problem. Van Dalen^^ and McGrath, Jelinek, and Wochner de- 
scribe problems as either (1) the adaption of a means to an 
end, (2) the lack of understanding of the character of an ob- 
ject or event, or (3) the existence of an unexpected event. 

Clark, Cuba, and Smith^^ use different terminology and add a 
category as they indicate that a researchable problem is an 
anomaly, an uncharted area, an unverified "fact," or the exist- 
ence of conflicting evidence. It is here proposed that the 
establishment of a problem has been accomplished if the researcher 
documents the existence of one of these situations. 

‘ Throughout the research process literature there is a con- 
cern expressed for the lack of an integral role for theory. The 

46 

importance pf theory has been most eloquently stated by Platt 
as he indicates that a major difference between those sciences 
that make rapid strides and those that languish in their data 
is the use of "strong. inference." This he indicates evolves 
from the theoretical construction of a. logical inference tree. 



B. Van Dalen, o£. cit . 

^G. D. McGrath. J. J. Jelinek, and R. E. Wochner, o£. cU. 
^^0. L. Clark, E. G. Guba, and G. R. Smith, o£. £it. 

R. Platt, 0 £. £it. 
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a tree where the forks of the branches represent alternative 
solutions to problems impeding man 's progress or knowledge. 

The systematic testing of these alternatives adds, to our 
understanding. The failure to use some guiding conceptual 
framework leaves us collecting data which has unknown rele- 
vance . 

The appropriate question to facilitate research evalua- 
tion is how does one build the conceptual framework or theory 
so desired in research? It-is here believed that this is 
initiated through the activities of identification of (1) 
the variables known and/or suspected to be operant in the 
problem area, (2) the relationships among these variables 
again including both the known and suspected, and (3) the edu- 
cational, social, and scientific antecedents of the problem 
situation. The theoretical base or conceptual framework is 
completed when the researcher is able to structure and state a 
set of assumptions which will enable him to conjecture as to what 
is and where in the scheme of this is the problem. 

it should be pointed out that in the last sentence the 
word 'bssumpt ions , " plural, was used. These assumptions are the 
focus for research, for the advancement of our science requires, 
the movement of a point from the category of assumption to the 
category of fact. That is, our end is being able to know what 
variables are involved rather than accepting their possible 
involvement— knowing the relationships of these variables rather 
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than accepting the possiblity and nature of their relationship. 
The acceptance of this point establishes the need for the specif i 
cation of the objective to be achieved in a given study for two 
objectives, to identify variables and to test relationships are 
implied each of which may be subdivided into an array of goals. 
The final criterion, the statement of the limits within 
which the study is conducted, relates both to the theory woven 
about the problem and to the setting in which the problem is 
studied. As implied in the above discussion, the hypothesis to 
1^® tested in a study is derived from one or more of the assump- 
tions in the theory. The existence of other assumptions sets 
limiting conditions on the study which must be considered as the 
research progresses. It is also possible that the site of the 
test of the hypothesis, both in terms of time and physical char- 
acteristics, provides some limits to the absolute solution of 
the problem. Thus, their idei>tif ication is mandatory as an aid 
to the reader's interpretation of the study. 

In closing this discussion on the evaluation of the problem, 
attention is called to the absence of a criteria of significance 
or justification for the research. This omission is by design 
for it is here believed that the establ ishment of the existence 
of a problem and the structuring of a theoretical framework des- 
criptive of it provide sufficient justification for its study. 



o 
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Evaluative Criteria for the Hypothesis Component 

Writings on the role of hypothesis in research project an 
almost human capacity to something that is little more than a 
collection of words. For example, one can find statements that 
hypotheses 

!• ... provide direction to research. 

2. ... prevent the review of irrelg,yant 

literature or the collection of useless 
data.^ 

3* ... sensitize the investigator to certain iq 

aspects of the stiuation which are relevant. ° 

... is required to provide a framework for 
stating the conclusions in a meaningful 

manner. 50 

5* ... serves as an intellectual lever by 

which investigators can pry^ loose more 
facts to be fitted into other more con- 
clusive explanations.^ 

Skipping through these statements conjures up the picture of a 
little genie that appears magically and whispers in the investi- 
gator's ear, "Don't read that study. It's irrelevant."; Who 
grabs the investigator's pencil and shouts, "Don't record these 
data. They're useless!" Then, by magic, the hypothesis genie 



47 

48 



G. J. Mouly, ci t . p. 89“90. 
Ibid. 



49 



ibid. 



50; 



51 



D. B. Van Dalen, o£. £i_t. p. 156. 
Ibid. 
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alters his form to becone a solid steel skeleton upon which 

flashing neon conclusions are fastened. Again an alteration, 

and our faithful hypothesis Is the longest and strongest of 

crowbars. Would that researchers could find such a dandy com- 
pan ion. 

Enough of the dreaming; if one Is to evaluate hypotheses, 
.It IS imperative that he know what they are as well as what 
they are not. Definitions of the term range In sophistication 
from Hlllway's statement that a hypothesis is "... a reason- 
able guess or supposition based upon the evidence available at 
the time the guess is raade,"®^ to Cuba's statement that 

Within the framework of a theory, hypoth- 
eses are deductions following from and 
logically consistent with the assumptions 
on which the theory is based. 53 

The statistician provides a different focus in stating, "Hypoth- 
eses, whether statistical or research, are usually concerned 
either with differences or deviations. This writer prefers 
Kerlinger's attempt at synthesizing all of the above as he de- 
fines a. hypothesis as . .. a conjectural statement of the re- 
lation between^ two or more variables . "55 

Z_ . 

Tyrus Hillway, Introduction to Re sea rrh- Second Fdiftnn 
Boston: Houghton Mifflin Company, 1964. p. 123. 

53 

_ , Egon ’ G . Cuba, "The Writing of Proposals " in Resear>r*h ir 
Ed ucational Administration , edited by Stephen p! HencHy. 

54 . 

Jnhn Mythologic al Statistics . New York: 

John Wiley and Sons, Inc., 1 9557 p. 6i. 



55 



F. N. Kerlinger, o£. £i_t. p. 20. 
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With this definition of a hypothesis let us return to 
the magical claims In the riterature. |t should be patently 
clear that a statement Is unable to direct, prevent, sensitize, 
etc. It should also be clear that these are necessary aspects 
of research. That Is. the research must have direction. The 
researcher must classify and categorize the Irrelevance of data. 
He must establish a framework for conclusions. The hypothesis 
stated In a research Is only the mode used by the human to state 
those aspects he has worked through. 

Accepting this argument, four criteria seem relevant to 
the task of evaluating a hypothesis. 

1. Does the hypothesis state or directly -a 
I mply the existence of two variables?^® 

2. Does the hypothesis state or directly 
Imply a relation between the varlables?^^ 

3* Are the variables empirically observable?^^ 

Is the hypothesis based In a theory or a 
body of previously established knowledge?59 

Point 4 In the above list bears some elaboration. One 
contributor to the snail-like rate of progress due to educational 




58 « 

e. J. Mouly. £lt. p. 92. 
^^Ibld. p. 91 . 
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research is the tendency to empirically approach a single hypothesis. 
In contrast the physical scientist typically starts out with sets of 
hypotheses vfhich he systematically works his way through. He seems 
to ask himself, 'Vhat could have caused this?" and answer hypotheti- 
cally, "It could have been A, or B, or C . . . ." The test of a 
single one of these hypotheses invites- little or no advancement 
through a no-s ignif icant-difference finding or the failure to identify 
multiple causation. Thus, the failure to base the evolution of a - 
research hypothesis either in theory or substantial body of knowledge 
from which rival hypotheses can be or are evolved reduces the effect- 
iveness of a given study. 

m 

m 

This set of criteria rejects one item frequently found in 
the literature. Nouly enunciates this one clearly as he states, 

"A good hypothesis must be stated as clearly and concisely as 
the complexity of the concepts involved will allow."^® This 
seems to structure the evaluation of research on the matter of 
literary style rather than on actions in the research process. 

The potential research evaluator ought to ask at this 
point what about the case in which a hypothesis is not ex- 
plicitly or implicitly stated. Do we reject as research the 
situation in which a concern focuses our attention on an area 
in which the quantity and quality of existing knowledge precludes 
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hypoth.,I*I„97 The -Wherted ere... probI«, category 1, . 
point. ,f one consider, . f r.,d 

-rk ha. been done, it I. u„Ute,y *hat he is able to specify 
the bonstroct. that «y hypothetically explain the phen<x«ena. In 
this sitoation he needs infonnation which woold describe the n-ber 
n.to,e. „d ™„tionship of these concepts, m other words, he 
-nts to know What are the variable, that are involved, what is 
their natore. and what are the relationship, between variables. 
this respect ,«stions can be «„loyed to give .y„ection to a study 
•ipfvent the collection of irrelevant data," -provide a fraeework 
for conclusions... and so on through all the clai« „de for hypoth- 
«»•*. Thus, the criteria for evaluating guestions should include 

ure or variables m a given problem? 
the variable in each question observable? 

o? ““*“"9 "ooy 

The presentation of these criteria for evaluating ques- 
tion. in research speaks directly to PIatt.,61 eorw^rn for 
.eking the .^crucial question., by stating that a question should 
oUher seek the identificet.on of variable, „r their description. 

-t further propose, thet if enough is known about a problem that 
archer can conjecture on the existence and relation of 
variables, a hypothesis is warranted. 



2 . 

3 . 




J* R. Platt, og. cit. 
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Evluativ Crif ria for th« itevi<w of Related Literatuf^^ 

In tha discussion of both the problem and the hypothesis 
or question components above, definite Implications for the re- 
view of literature have been enunciated. Thissegment will 
attempt their explication. 

The Individual who reads quantities of research reports 

frequently Is In agreement with Lindvall's^^ Judgment that 

all too often the review Is not an Integral part of the study. 

It is here proposed that this difficulty Is a direct result of 

6b 

the general trend Lake'^ finds, i.e., that the majority of 
researchers are raw empiricists in contrast to hypothetlc- 
deductlvlsts. To the latter, knowledge is cumulative. Thus, 
the use of what Is known to set the theoretical frame%«ork 
gives meaning and relevance for a related literature review In a 
report. Fatiure to 'see such a purpose makes the review almost 
an academic task of producing a lengthy annotated bibliography 
and/or proving the uniqueness of his study. Both are rejected 
by Llndvall^^ as central to the study. 

62 

Much of this discussion Is adapted from the Clark', Cuba, 
and Smith outline; og. cit . 

N. LindvaU, 'Heview of Related Research." Phi 
ielte Kappen b O: 179-80: January 1959- 

64 

K. E. Uhs, OP. cIt. 

^C. N. Lindvall, og. cit . p. I 79 . 
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The purpose of reviewing the litereture then is for the 
deveiopnent of e theoreticel or knowledge base upon which the 
substance and methodology of a given study are built. Criteria 
for evaluating this' achievement have already been expressed 
under headings above. There are, however, specific activities 
involved that should be elaborated here as the basis for estab- 
* 1 ishin^ specif ic criteria. 

If the researcher is to facilitate growth on the part of 
any audience, he can do so by helping them to know the setting. 
Thus, by listing the extent of the review and the specific 
bibliographic references found relevant, the researcher contri- 
butes to the definition of the setting in which the problem is 
studied. 

To accept as complete, stopping with mere listing of re- 
lated literature or providing what Mon roc and Englehart call a 
"classified annotated bibliography,"^ is to declare this entire 
paper and much of the focus of this conference as unnecessary. 

The differential value of various researches has been substantially 
\ documented. Thus, if a review is to be of vdlue in building a 

theory of knowledge base, the strengths and weaknesses of each 
article included must be identified. 

W. S. Monroe and M* D* Engelhart, The Scientific Study 
of Educational Problems . New York: The Hacmi 1 1an Company ,1|^6. 
p. W. 
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The final activity for evaluating the review may relate to 
the manner in which the problem statement is presented and the 
design is structured just as well or better then under a separate 
rubric, related literature. This activity is the synthesis of 

what is and is not known about the subject. 

Each of the items for evaluating the review also enunciate 

the review's topical emphasis. Here two areas are proposed: the 

substantive area, and the methodological area. Through the review 
the individual should be presented with a synthesis of what is. and 
is not known about the subject at hand ^ about the proposed method 

for studying that subject. 

The specific criteria could be stated as follows: 

1 Does the research report present a list of the 

studies completed in both the substantive and 
methodological aspects of the problem? 

2. Does the research report present a critique of 
the studies listed? 

3. Does the research report include a synthesis of 
idiat is known in both the substantive and 
methodological aspects of the problem? 

Evaluative Criteria for the Design 
The discussion under this rubric will be restricted to 
the definition of the term and brief recognition of some criteria 

^^D. L. Clark, E. G. Cuba, and G. R. Smith, o£. cjjt. 




applicable to all methodologies. This discussion wiii be au^nented 
in a iater section devoted to criteria for evaiuating research tech- 
niques specific to the study of change. 

Neither those engaged in research nor those observing their ‘ 

work attempt to refute Barr, Davis, and Johnson's statenmnt that 

Educationai research is a compiex activity; 
oniy through the most meticuious specifica- 
tions can the many factors that need to be 
kept ^ mind be controiied at the proper 
time.^ 

Thus is the justification of the design component of the research 
process. 

Design in this context is defined as that pianning in which 
the researcher engages to insure the accumulation of the most power- 
ful conclusions about the nature of the problem. Journals have long 
conveyed the assumption that a consumer should be informed of 
these plans through the inclusion of a procedures or design 
section in their format. 

Lindquist presents a cogent statement in describing the 

ingredients of an experimental design. 

The important decisions to be made in 
planning the experiment are concerned 
with: (1) the definition of the 

"treatments," (2) the selection and 
exact definition of the population to 
be investigated, (3) the selection of 



68 

A. S. Barr, Jl. A. Davis, and P. 0. Johnson . . Educational 
Nesearch and Appraisal. Chicago: J. E. Lippincott Company, 1953* 

p. 309. 
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a criterion, (4) the identification of 
the factors to be controlled, (5) the 
final restatement of the problem, and 
(6) the selectj^n of a specific experi- 
mental design.^ 

If broadly interpreted, riiany of these are applicable to descrip- 
tive and historical methodologies. For example, items (1) end 
(2) in this list are important to both historical and descrip- 
tive research. The historian is interested in determining 
what is the pattern of events (or treatments) and the strength 
of their contribution to an historical event. This is docu- 
mented by Travers as he states, 'Historical studies usually be- 
gin with a delimitation of the general category of events that 

70 

IS to be reconstructed." The descriptive researcher is inter- 
ested in the status of a particular group at a given time. The 
reason for his interest, that is the pattern of events or 
circumstances which have made this group the focus of his 
interest equates to the experimental term "treatment." 

Another example of the extension of Lindquist's ingred- 
ients of design is in the area of criterion selection. In any 
research, descriptive, historical, or experimental, the decision 



69 

E. F. Lindquist, Design and Analysis of Experiments in 
Psychology and Education. Boston: Houghton Miff 1 in Company, 

1953. p. 7. 

M. W. Travers, An Introduction to Educational Research; 
Second Edition . New York: The Macmillan Company, 15164, p. II 5 . 



o 

ERIC 



28 



mu,t be ,»da regerdir-s whet !. .cc.ptrt.Ie evidence either for 

testing the hypothesis or ensuring the questions rel.v«,t to 
the problem. 

The specific decision points upon which research planning 
focuses are the population, the s«»ple, the variables involved, 
the control, necessary, the data collection techniques, and thi 
enalysis procedures. Previous sections have discussed the Justifi- 
cation of the focus on v.ri.bie. and their relationship. ,f through 
researeh w. attempt to make statements with any general ity, our plans 
must mclude a careful focus on population. What a « the character- 
istics of the units in the popul.tion(s)? ffhe parenthetical plural 
1* «tr«»|y i„port.nt in some studies, e.g., „cNel,7' ,,,, 

•-has a population of students ^nd a population of teachers.) with- 
out carefu, definition her. the .eaves no for appMc 

billty. 

Sampling in a given population not only enables the researcher 
.,to focus upon a group in which he can efficiently conduct his study 
but also has relevance to the analytic., model. Certain smnp.e 
characteristics support the N.yman- Pearson model, others upport 
a Bayesian approach. ,t is of iitti. value here to debate the ade- 
y Of these theories as such requires far more accomplished statis- 
ticlans than,!. However, It is Important that we focus on the need 



1:113-111; March ; ^*964. ”” ’ g<‘ucatlonal ae...„K 



for Information about the sample($) utilized In order that the de- 
sign might be evaluated. 

The Importance of the data collection technlque(s) In a study 
are penerally accepted. If a researcher Is to make an Inference re- 
garding the truth of his hypothesis, he must collect observational 
,.data on suspected consequences. The objectives In aqy deta collec- 
tion technique are •■. . . to provide accurate observation, to eli- 
minate observer bias, and to extend and quantify the observations of 
the human researcher.»72 jhree concerns seem Imperative. Are the 
techniques valid for measuring the consequences predicted? Is there 
consistency In the measurement? And finally, are the techniques ob- 
jective? The general acceptance of these points Is so great that It 
was a source of amazement to find that criteria of Instrument reliabll 
Ity and validity did not discriminate between good and poor research 
reports In Gephart's study of a research evaluation Instrument.^^ 
Although procedures for analysis of the data are decisions 
inherent in the design, their importance as an aspect of research 
warrants their treatment as a major component of the research process 
rather than being subsumed under the heading of design. 
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Given the above, general evaluation of the design rests on 
the adequacy which can be attributed to the anwers to the fol- 
lowing questions. 

1. Does the research report define the population of 
people, things, or occurrences inherent in 

the problem? 

2. Does the research report describe the sample selec- 
tion procedures and/or the characteristics 

of the sample? 

3. Does the research report operationally define the 
variables studied and the variables known to be 

. associated in the problem? 

4. Does the research report describe the controls em- 
ployed to counter the effects of the latter group 
of variables? 

5. Does the research report specify optimally valid 
and reliable data collection devices or techniques? 

\ 

Evaluative Criteria for the Analysis of the Data 
Systematic analysis of the accumulated data is imperative 
in order to determine inherent facts and meanings not necessarily 
apparent in casual examination. A negative example makes the 
point. asked counselor educators and teachers to rank 

the importance of forty^one tasks performed |>y elementary school 

counselors. His observation that the relative importance attachecj 

✓ 

to several of the tasks by the two groups differed greatly led 



74 

R. N. Hart, "Are Elementary Counselors Doing the Job?" 
The School Counae I or 9:70-2; December 1961. 
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him to conclud. that eounsolor. could not m.k. both sroup* happy. 
Had ha calculated a correlation batwaan the two rank* he would 
have found agraamant (Rho - . 796 ) significant at the .001 laval. 
Rdthar than hi* pessimistic conclusion, ha should have acknow- 
ledged the significant agreemant among the two groups. 

Stanley.*?? d„cusslon of research reports In Volume I of the 

Aar lean Educational Research ,|o„rnal Identifies additional cases 
of infllyticfll inid6(|UAc {os . 

Best has stated the outcomes of the data analysis. 

ai^ significant conclusions are derived. 

These conclusions will be based upon comoarl- 
“n™'orl™Je”:78^ relationships of one” 

Thus It would appear that In the analysis of the data a statis- 
tical description of the data and the statistical significance 
of those data are vital. 

Much has been written which specifies which statistical 
procedure should be employed on a given set of data. Three 
notable attempts at synthesising this 1 Iterature provi* asslst- 
ence in determining which stetistic is appropriate. These can 



tlon.-.^*ii„e'og“S;ir;ai''I"ri: 5 T 2 "‘ f “'^'cnal Experiment. 
Symposius, Madison. Huion.™! 1^5 *' 
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be found io the tables constructed by Senders, Siegel, 7^ end 

79 

Tetsuoka end Tiedemen. In each case a grid has been con- 
structed through which the appropriate statistic can be identi- 
fied by determining the number of variables involved in the 
analysis, their scalar nature, and the relationship between the 
samples (dependent or independent samples). 

Given the above discussion, evaluation of the data analysis 
in a given study rests on the nature of the answers to the fol- 
lowing questions. 

1. Does the research report systematically organize 
the accumulated data? 

2. Does the research employ appropriate statis- 
tical procedures in analyzing the data? 
(Appropriate herein is defined by the scalar 
nature of the data and the design employed.) 



Evaluative Criteria for Conclusions 
Many have expressed concern for what goes into the con- 
clusion component of the research process. This concern ranges 
from the contents of the conclusion statement to its form. The 



L. Senders, Measurement and Statistics . New York: 
Oxford University Press, I9S8. 

78 

' S. Siegel, Nonoarametric Statistics for the Behavioral 
Sciences. New York: McGraw-Hill Book Company, Inc, 1^56^ 

79 

^•'M. M. Tatsuoka, 0. V. Tiedeman, "Statistics as an Agent 
of the Scientific #ithod in Research on Teaching," in Handbook 
Qf Reaearch on Teachlne. N. L. Gage, Editor, Chicago: Ran3 
McNally and Company, 1^3. 
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first is exemplified by Hi llwey's statement regarding the kinds 

pf statements that might be admissible. 

These are: (1) a basic assumption, (2) a 

statement of fact, (3) the writer's opinion, 
and (41.the opinion of an authority in the 
field.®® 

m 

The evolution of conclusions would be facilitated if each of 
these types of statements is clearly identified. Van Oalen's 
words attach the evaluation of conclusions to the plausible 
logic inference pattern described earlier. He indicates that 
a conclusion cannot be drawn regarding the truth of the hypoth- 
esis until 

... it meets all of the following re- 
quirements: (1) all the factual evidence 

collected in the empirical tests corresponds 
with the consequences (of the hypothesis); 

(2) the test situation adequately represents 
the essential factors expressed in the con- 
sequences; and (3) the consequences are logic- 
ally implied in the hypothesis."^ 

Synthesizing these three statements defines a conclusion as a 
statement about the truth value of the hypothesis given the condi- 
tions of the specific study. 

If a research is to contribute to making cumulative the body 
of knowledge or to the evolution of theory., the research should pre- 
sent a statement of implications. That is, if in the analysis of a 



«®T. HillMy, cit. p. 137. 

0. B. Van Oalen, cit . p. 139. 
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specified set of dete^ support is garnered or lost for e given hypoth 
•sis, then e discussion which speaks to the meaning of this conclu- 
sion for our professional growth is imperative. Does this specific 
study strengthen certain theoretical assumptions, or does it imply 
modification in the theoretical base and suggest needed research? 

Given acceptance of the above statements, the following 
questions are posed as criteria for evaluating the conclusion com- 
ponent of a research report: 

1. 0ms the report sute whether the findings 
firm or disconfirm the hypotheses? 

2. Does the report state the conclusions drawn 
from the findings? 

3. Are the conclusions drawn from without qoinq 
beyond the data? 

0ms the report describe implied modifica- 
tion in theory raised by the conclusions? 

5* Does the report state specific problems 
raiMd by the investigation that require 
add i t ional research? 

ELEMENTS OF THE STUDY OF THE EDUCATIONAL CHANGE PROCESS 
To set the stage for a discussion of criteria specific to 
the assessment of methodological adequacy it is believed Mcessary 
to attempt a definition of the field for such research. The 
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synthesis of statements by Rogers^^ and Walton^^ provides the basis 
for this definition task. 

Rogers' Mork highlights three variables inherent in the 
study of change, the innovation, the target unit, or that which 
is to be changed, and the' initiating unit or change agent. In 
the discussion which follows these variables will be included 
under the rubric "actor variables." 

As one examines research on change it becomes obvious that 
there are interactions between these actor variables. That is, 
an innovation with a certain set of characteristics is more 
acceptable or less acceptable to target units of different character. 
A change agent of one type may be effective with one innovation and 
not with another, or with one target unit and not another. In 
statistical language, the field of study focuses on the description 
of,. and the assessment of, the main and interaction effects of the 
three actor variables. 

A second set of variables, "action variables,", seems implied 
as Rogers also describes an adoption process or an action sequence. 
His presentation seems limited, however, as it is designed from 
the target unit vantage point. As such, it implies but also tends 



E. Rogers, Diffusion of Innovations . New York; The Free 
Press of Glencoe, 19^2. ^ 

^^R. E. Malton, "T««o Strategies of Social Change and Their 
Dilemmas." The J^rnal of Applied Behavioral Science. 1:167-79; 
Apr 1 1 -Nay-June 19^. 




to obscuro tho concopt of action on tha part of tha chanQa agant. 
Malton's discoursa highlights tha intaraction batwaan two changa 
agant action variables. Thasa ara actions ancompassad in aithar 
a poMr strategy or an attitude strategy and his discussion of them 
is thought provoking. The purpose here is not to question thasa 
strategies as tha only possible actions but to highlight their 
existence and intaraction. Tha description of these action var-' 
iablas and tha assessment of their affect of the change process, 
either individually or in concert, provides a second focus in the 
definition of the field of study. 

Just as there is suspected interaction between actor var* 
iables and between action variables, so too there should be sus- 
pected interaction between actor and action variables. That is, 
change in a given target unit may best be accomplished through a 
given action. Or, change may be facilitated if a specif ic .behavior 
is employed in a situation involving an innovation- target unit 
interaction. 

Given the above points a four dimensional model can be con- 
structed. As four dimensionality is impossible to picture graph- 
ically, the following build up of the model is presented. It is 
possible to conceive of the three actor variables as each contri- 
buting a main effect to the process of change in an educational 
institution and their contribution through two or three way inter- 
actions. Thus three axes in the model are set by the actor variables 
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(Y) 

Any given innovation may contribute x units to the an»unt of change 
that occurs in the system. A specific target unit, a school, an 
educational level or discipline, a specific teacher contributes y 
units. (Here may be the propitious point to insert the possibility 
of negative change.) Finally, the change agent adds (or detracts) 

2 units of change due to his nature, position, relation to the tar- 
- get unit, etc. Point A then represents the amount of change ex- 

• pected in a system from the static existence of an innovation, a 

target unit, and a change agent.** , 

% 

When the action variable— that is, a power strategy of legis- 
lation, remuneration, etc., or an attitude strategy of interpersonal 
involvement, education, etc.,— is inserted into the study of the change 
process, a fourth dimension is needed. In other words when an inno^ 
vatton, provision of counseling for students, is offered to a target 

♦ 

o 

me 



unit, th^ secondary school, by a change agent, a professional organ* 
ization, change-of differing quality and quantity will result from 



*^^^^***^^9 actions, legislative lobbying versus educating teachers as 
to the need for such a service. The inclusion of ail four factors and 



their interrelationships can be affected through a four-dimens iondi 

I 

model in which the actor variables account for three of the dimen-l 
sions and the action variables, the other. Such a model focuses ' 
our attention on the following: 

1 . 



The actor variables: What are they and what i^ar- 

iance exists on each? | 



2 . 



The action variables: what actions or activ 
are involved and again what is the variation 
possible on each? 

The interrelationships between actor, action, 
actor-action variables: What interreiationsh 



ties 



and 

Ips 



exist? What are the effects of one upon another? 

* i 

The researcher can interpret this model in a manner which suggesU 
research activities. First, it seems imperative that the variabjies 
be identified and that their characteristics be understood. If we 



} 

are to build a model of change, we must know its constituents. 'Sec- 
ond, if our model is to become a theory, we must know these con$tit- 
uents well enough to at least conjecture about their relationships. 
Thus it iniould seem that historic and descriptive studies are iilport- 
ant in setting the model and that experimental studies are valuable 
in testing the relationships that exist. 




METHODS AND TECHNIQUES FOR STUDYING THE 
CHANGE PROCESS COMPONENTS 

The terms "research methods" and "research techniques" are 
too often used interchangeably in the iiterature. In the dis- 
cussion which foiiows, 'Vnethod" refers to the . investigatory strate- 
gies: historicai, descriptive, or experimentai study; while "tech- 

niques" refer to the specific actions taken in a given study. This 
iatter area inciudes techniques of sampie seiection, treatment, data 
coiiection, and data anaiysis. 

The description of the three research methods iisted above 
typicaiiy pieces them on a continuum of time. For exampie. Best 
states, 

Historicai research describes what was. 

Descriptive research describes what is. 

Experimentai research describes what wiJU 
be when certain factors are controi ied.®^ 

A second dimension for categorizing research methods is presented 

by Best in the same discussion. In this he discusses the techniques 

typicaiiy empioyed in each method. For exampie, the historian 

attempts to identify "primary, originai, or first-hand sources of 

85 

information," for the purpose of understanding change. The 
descriptive researcher engages in the accumuiation and anaiysis of 
data for the purpose of describing status. 
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Th6 morgar of th6 abov6 two dimensions for defining research 
methods induces confusion. If one is interested in the extent to 
which a specific innovation^^iet us say providing counseling oppor* 
tunities in high schools— had been adopted in 1950, Iw is attempt- 
ing to go back into the past to determine what was. Further* he 
must seek out first hand sources of information to conduct a valid 



study. At the same time, he is accumulating and analyzing data to 
describe status. Is he engaged in a historical or descriptive 
study? An ^ Rpst facto e xperimental design presents some of the 
same conflict. Case studies also add to the confusion. 

It is here proposed that, rather than a conceptualization of 
research methods as categories, greater clarity may be obtained 
through analysis of methods according to the amount of control an 
investigator has in generation of data. Three factors, the units 
or subjects involved in the population and sample, the treatment (s) 
of these units, and the data accumulation techniques employed, 
structure the generation of data. Thus research method exists in a 
cubic model as shown below. 

F G 




H 



Observational 

technique 



er|c 
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In this conceptual scheine historical research exists at the 
lower left hand corner (^) of the three axes. Here data was gener- 
ated through unspecified observational techniques on an unselected 
set of units which experienced some uncontrolled treatment (s) . The 
true experiment involves the selection of subjects on a random basis, 
careful control over the treatment, and the selection or construc- 
tion of valid and reliable observational techniques. Thus the true 
experiment exists at the opposite corner of the cube (G). Quasi- 
experimental studies as described by Campbell and Stanley^^ exist 
when the investigator has control over the treatments and the observa- 
tional techniques employed but lacks control over the units involved. 
As such quasi -experimental studies are located on the ABFE face of the 
cube. The "better" the quasi -experiment, i*e., the greater the con- 
trol over treatment and the more valid and reliable the measuring 
technique, the closer the study is located to point F. 

The descriptive research method omits control over treatment. 

In such a study, the researcher has the power to structure the in- 
clusion of units and to utilize valid and reliable observational 
techniques. Thus the descriptive study locates somewhere on the 
AEHD face of the cube. Again the "better" the study, the closer it 
may be located to a corner of the cube, in this case H. 




Experimental 
on Teaching. 



CapbeM and J. C. Stanley, "Experimental and Quasi- 
Designs for Research on Teaching" in Handbook of Research 
N. L. Gage, Editor; Chicago: Rand McNally, 1963. 



Conceptualization of rasaarch methods through this cubic 
model faciiitates the understanding of the cohtrubution of each 
method. Perhaps, in iight of this, v«e might avoid the frequent 
vaiuing and devaiuating of research by methodoiogicai types and 
base our Judgments on the adequacy of the specific techniques em- 
pioyed in a given study. 

* * - yJ 

The appiJeation of each of these methodoiogies can con- 
tribute signif icantiy to knowiedge of the process of change in 
education. For exampie, information about the process of change 
couid be gained through historicai study of the spread of an 
innovation; descriptive studies on the nature of the innova- 
tive activities currently found in educational institutions 
also are valuable. In essence this is the methodology employed 
by Mcrt and his associates. Ascertaining the effect of one 
variable on another, the experimental method provides a sigr . 
nificant means for developing understanding and making predic-* 
t ions regarding the action-actor variable interactions. Thus, 
the adequacy of the several research methods for the study of 
the process of change depends on the objectives of the investi- 
gation. 

The term "research technique" here means those activities of 
subject selection, treatment administration, and data collection, 
evaluation and analysis employed in a given study regardless of 
general research method employed. An examinat^n of the research 

87 

reports included in Roger's bibliography identifies some variety 
in techniques in research On change. Sample selection techniques 
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employed Include random selAri-tn» « 

• population .nd .celd.nt.1 

PPloetlon through the use of Intect c. 

t groups. Since most of this 

-pprch „ historic, or des.,pt.e. there see. to he . .crc- 
ty Of technlgue, for e*„n, storing treetnents. For the p.,. 
- s-th.pg heppened to , group hnd It we, studied ^ 

-V with on., .n |„p|, 0.t. collection see™. 
-St condom, to proceed through Interview .trough 

P.rt.clp.„t observer technigues. Se.don, Is there a discussion of 

the velldlty or rellehl.lt, of the obtained date. Descriptive 

Statistics includina oer r^m- «« 

9 pe nt of response seem to be the most 

common data analysis in • 

y In some instances correlational and factor 

' .d^lttedl, restricted to 

-9er.sblbllograph,.lndlcates that Investigators of change have 

o o^d In a tradition of technigue that Is perhaps limiting real 
• vancement In our understanding and control of change. 

The evaluation of technigues emplo,ed In a specific research 

7 " ‘hP specific tech- 

"•dues empl.ced and the appropriateness of each technigue In the 

rPsearch strateg,. ,1. prohibits tl» enunciation of criteria for 

«ac possible technique. General ized criteria fu 

aiizea criteria on these techniques 

include those stated earlier in mx 

•Tiler In discussion of the design and data 

anal,sls research components ir ... • 

. • ""PP^ant here to emphasise 

expand on some of the alread, stated criteria. 
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Campbell and Stanley'^ present a generalized discussion of 
design in which two concerns are proposed, internal and external 
validity. Their suggestion focuses heavily on the experimental 
methodology. However, certain aspects have relevance to other 
methodologies. 

Internal validity, that degree to which the study tests what 

it purports to test, depends upon the control of eight mediating 
89 

variables. These contributors to internal invalidity are 

1. History--the specific events occuring be- 
tween the first and second measurement or 
simply prior to a posttest in addition to 
the experimental variable. 

2. Maturation— processes within the respond- 
ents operating as a function of the passage 
of time per se (not specific to the partic- 
ular events), including growing older, 
growing hungrier, growing more tired, and 
the 1 ike. 

3* Testing--the effects of having taken a pre- 
test upon the scores of a second testing. 

4. Instrumehtat ion--changes in the calibrations 

of a measuring instrument or changes in the 
observers or scorers used which may produce 
changes in the obtained measurements. 
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5- Statistical Regrass ion— operating Mhere 

groups have been selected on the basis of 
their extreme scores. The tendency of lower 
scorers on a test to score higher on a re- 
testing due to the presence of measurement 
errors. Also tiie tendency of high scorers 
to score lower on a retesting. 

6. Selection- biases resulting from picking 

different respondents for the comparison 
groups. 

7- Mortal ity - differential loss of respondents 
from the comparison groups. 

8. Selection - -maturation interactions, etc. — 
where two of the previous factors working 
together might be mistaken for the effect 
of the experimental variable. 

The sources of external invalidity are of equal concern as 
they restrict the applicability of the findings of a given study. 
Campbell and Stanley^® suggest four such factors: 

1. Interaction of testing and X— tdiere a pretest 
might increase or decrease the respondents' 
sensitivity or responsiveness to the experi- 
mental variable and thus make the results 
obtained for a pretested population unrepre- 
sentative of the effects of the experimental 
variable for the unpretested universe from 
vfhich the experimental respondents were 
selected. 

2. Interaction of Selection and X—the tendency 
of a. typical subject to seek out or volun- 
teer for a study, thus making the subjects 
different from persons in genciral. 



^Ibid . p. 171-^46 para. 
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3 . Reactive Arrangements— effects of experimental 
arrangements Mhich vfould preclude general iza~ 
tion about the effect of the experimental 

SrtInSS'."® 

4 . Multiple Treatment Effects— in studies which 
involve alternating groups among treatments, 
one may come up with conclusions applicable 
only where this specific sequence of events 

is possible. 

The relevance of these variables to methodologies other 
than experimental is demonstrated in the following examples. 

In a historical study, changes occurring within an institution 
may be maturational rather than the effect of some action. As 
a newly established school ages, the changing perceptions of each 
other on the part of the staff may be as important a factor in 

I 

a change as is the action of a change agent. Inst rumen ta ion 

changes affect both historical and descriptive methodologies. 

\ 

The historian idio analyzes several documents may be classify- 
ing or evaluating the last document on the basis of different 
criteria than he did the first. The survey employing an inter- 
view may i.iterpret the responses of the last subject in a differ- 
ent light than the first. 

The works of Barker, Rozenthal,^^ and the study of the 

iT ^ 

A Ik k The Sti ^am of Behavior . New York: . 

Appleton-Century-CroftT! 1963. 
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R. Rozenthal, *Hesearch on Experimenter Bias.” Paper 
rwd at American Psychological Association, Cincinnati, September, 
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HaMthorrw Effect** all relate to the react ive arrangement 
category of external Invalidity. When a person is engaged In 
an Investigation, he tends to behave for the study In contrast 
to normal behavior. Orne^^ indicates that study is needed to 
determine what are the demand characteristics of a study. 
Barker^** suggests a mode of study in which the investigator is 
not a participant but an observer In which the unaffected 
"stream of behavior** is recorded for analysis. The question 
vital here is, to what extent in a given study did the study 
of change affect change? 

Another area of concern under the research technique 
rubric is the concern for sample selection techniques. The 
adequacy of a study of adoption of liew staff utilization tech- 
niques in schools rest heavMy on the extent to which the 
schools in the study represent schools in general. Cornell 
and McLoone^^ in reviewing the design of sample surveys in- 
dicate that in sampling attention must be paid to a precise 
description of the population or universe and to a determination 

93 . 

Hart in T. Orne, *t)n the Social Psychology of the 
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of the permissible error and acceptable risk or confidence level 
to determine the appropriate sampling techniques. 

The third area of research technique that needs evaluation 

is the date collection techniques. Here four concerns are ex- 
pressed. 

1. Is the data collectioji technique a valid 
measure? 

2. Is the d»ito collection technique a reliable 
measure? 

3. What is the degree of objectivity of the data 
collection technique? 

What is the practicality of the data collec* 
tion technique? 

Although these items are listed in r^fative importance, the 
researcher finds he must forfeit here to gain there. Thus, 
assessing adequacy involves the search for optimal conditions 
of validity, reliability, objectivity, and practicality. 

The variety of data collection techniques is almost limit- 

f 

less. For example, the historian may examine records or documents 
or engage in interviews; the surveyer may observe, utilize 
standardized tests, projective techniques, interviews, etc. 

The experimentalist hps equally as broad a variety. Research 
on change in the past has incorporated the participant observer, 
the accumulation and analysis of records, testing, surveying, 
and interviewing. 

The determination of the adequacy of a given technique 
requires the assessment of the appropriateness of the data so 



collected for testing the hypothesis or answering the question. ■ 
Secondly, adequacy rests on the assessment of the validity- 
reliability-objectivity-practicality of the measuring device. 

In assessing instrument adequacy of the evaluation of research 
must (I) understand the types of validity and reliability and (2) 
interpret the importance of these for a given study. Under the 
rubric validity four categories exist. 

1. Content validity— the degree to which the 
Items in the measuring device are contained 
in treatment of which it is a measure. 

2 . '“"ffuet validity-the degree to which the 

instrument accumulates evidence on some 
hypothetical construct. 

3. Concurrent validity-the degree to which the 
instrument accumulates data which correlated 
with concommitant performance. 

Predictive validity— the degree to which the 
instrument accumulates data tdiich correlates 
with performance at some future date. 

It IS clear that, although we should be concerned with each of the 

above, for a given study one may be more important than another. 

Our concern for the reliability of measuring devices takes 
three forms; 



1 . 



2 . 



Internal consistency — the degree to which 

all parts of an instrument are measuring 
the same thing. ^ 



Stability— the degree to which subsequent 

^istrations of the instrument accumulate 
comparable data. 
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3. Equivalence*— the degree to which alternate 
forms of an instrument accumulate compar- 
able data. 

Again, although each is a vital concern, the relative importance of 
these items is determined by the specifics in a given study. 

The failure of a researcher to attend to both of these areas 
in his study and to communicate this in his report is severely depri- 
cating to the adequacy of the study. Without information of this sort 
we know not what we have measured nor to what extent we would obtain 
the measurement again. 

The fourth and final area of concern regarding research 
techniques is the area of data analysis. Through the liter- 
ature admonition can be found that appropriate and modern 
analysis techniques must be used. The definition of "appropriate" 
can be found in the study of assumptions inherent in statisti- 
cal models. "Modern," however, is an undefinable term and thus 
its use as a criterion is difficult. It is assumed that in any 
given report the analytical technique employed was as modern 
as possible for the investigator. In light of newer analytical 
techniques, the evaluator of research may infer weakness in any 
given study. 

The appropriateness of a statistic is based upon (1) the 
scalar characteristics of the data, (2) the number of variables, 

(3) the relationship among the variables, and (4) the manner in 
which randomization is inserted in the data generation. Senders^^ 

L. Senders, op . c i t . 
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presents a clear description of the four categories of scales, 
nominal , ordinal , Interval , and ratio. An argument rages as 
to whether statistics based upon interval assumptions can be 
used with ordinal data. |t can be demonstrated that some kinds 
of knowledge can be gained through averaging ordinal data. Per- 
haps a continuum of scalar qual ity exists. The crucial point 
seems not to be the employment of the incorrect statistic but 
rather the interpretation of the finding. The use of any 
mathematical process on frequency data presents findings re- 
garding frequency of responses rather than quality of response. 

The determination of the appropriate inferential statistic 
depends upon the number and the degree of independence of the 
variables. This is demonstrated in grid form by the presenta- 
tions by Senders,^ Tatsuoka and Tiedeman,^^ and Siegel. '0° 

CONCLUSION 

Several criteria for research adequacy, an attempt at 
definition of the field of research on educational change, and 



97 



97 ^ 

W. L. Hays, Statistics for Psychologists . New York: 
Holt, Rinehart and Winston, I 963 . p. 73 . 



98 



V. L. Senders, o£. crt. p. 256 - 7 . 



99, 



100 



M. M Tatsuoka and D. V. Tiedeman, o£. cit . p. 145-5. 

I 

S. Siegel, oj^. cit . (inside Cover) 



specific criteria for research techniques have been presented. 
Two points must be made in closing. To date, standards have not 
been drawn by which to measure the adequacy of a specific prob- 
lem statement, hypothesis, etc. The field seems to have settled 
on what must be done, and through this, apparently, agreement 
on research evaluation can be reached. Despite this lack of a 
final yardstick, research adequacy must be assessed if progress 
is to be made in the accumulation of knowledge or if the field 
is to be sure of that which is known. 

The second point relates to the general negativism that 
evolves from a systematic analysis of research. The Encyclopedia 
of Retrospect is a powerful book. Through hindsight we can 
observe all manner of error unobservable to foresight. However, 
without the action based upon limited foresight, hindsight is 
severly reduced. It is here proposed that Professor Designbumble 
did not set out to conduct a fallacious study. He did the best 
he could with the materials at hand and his level of sophistica- 
tion. Rather than berating his competence personally, the field 
will progress if the focus is on what was done right and- wrong, 
what was left undone, and what can be done to build on these. 
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